Author Response.

How to cite this article: Rahmatinejad Z, Hoseini B, Pourmand A, Reihani H, Rahmatinejad F, Eslami S, et al. Author Response. Indian J Crit Care Med 2024;28(2):183-184.

A comparative study of explainable ensemble learning and logistic regression for predicting in-hospital mortality in the emergency department.

This study addresses the challenges associated with emergency department (ED) overcrowding and emphasizes the need for efficient risk stratification tools to identify high-risk patients for early intervention. While several scoring systems, often based on logistic regression (LR) models, have been proposed to indicate patient illness severity, this study aims to compare the predictive performance of ensemble learning (EL) models with LR for in-hospital mortality in the ED. A cross-sectional single-center study was conducted at the ED of Imam Reza Hospital in northeast Iran from March 2016 to March 2017. The study included adult patients with one to three levels of emergency severity index. EL models using Bagging, AdaBoost, random forests (RF), Stacking and extreme gradient boosting (XGB) algorithms, along with an LR model, were constructed. The training and validation visits from the ED were randomly divided into 80% and 20%, respectively. After training the proposed models using tenfold cross-validation, their predictive performance was evaluated. Model performance was compared using the Brier score (BS), The area under the receiver operating characteristics curve (AUROC), The area and precision-recall curve (AUCPR), Hosmer-Lemeshow (H-L) goodness-of-fit test, precision, sensitivity, accuracy, F1-score, and Matthews correlation coefficient (MCC). The study included 2025 unique patients admitted to the hospital's ED, with a total percentage of hospital deaths at approximately 19%. In the training group and the validation group, 274 of 1476 (18.6%) and 152 of 728 (20.8%) patients died during hospitalization, respectively. According to the evaluation of the presented framework, EL models, particularly Bagging, predicted in-hospital mortality with the highest AUROC (0.839, CI (0.802-0.875)) and AUCPR = 0.64 comparable in terms of discrimination power with LR (AUROC (0.826, CI (0.787-0.864)) and AUCPR = 0.61). XGB achieved the highest precision (0.83), sensitivity (0.831), accuracy (0.842), F1-score (0.833), and the highest MCC (0.48). Additionally, the most accurate models in the unbalanced dataset belonged to RF with the lowest BS (0.128). Although all studied models overestimate mortality risk and have insufficient calibration (P > 0.05), stacking demonstrated relatively good agreement between predicted and actual mortality. EL models are not superior to LR in predicting in-hospital mortality in the ED. Both EL and LR models can be considered as screening tools to identify patients at risk of mortality.

Comparison of Six Scoring Systems for Predicting In-hospital Mortality among Patients with SARS-COV2 Presenting to the Emergency Department.

Background: The study aimed to compare the prognostic accuracy of six different severity-of-illness scoring systems for predicting in-hospital mortality among patients with confirmed SARS-COV2 who presented to the emergency department (ED). The scoring systems assessed were worthing physiological score (WPS), early warning score (EWS), rapid acute physiology score (RAPS), rapid emergency medicine score (REMS), national early warning score (NEWS), and quick sequential organ failure assessment (qSOFA). Materials and methods: A cohort study was conducted using data obtained from electronic medical records of 6,429 confirmed SARS-COV2 patients presenting to the ED. Logistic regression models were fitted on the original severity-of-illness scores to assess the models' performance using the Area Under the Curve for ROC (AUC-ROC) and Precision-Recall curves (AUC-PR), Brier Score (BS), and calibration plots were used to assess the models' performance. Bootstrap samples with multiple imputations were used for internal validation. Results: The mean age of the patients was 64 years (IQR:50-76) and 57.5% were male. The WPS, REMS, and NEWS models had AUROC of 0.714, 0.705, and 0.701, respectively. The poorest performance was observed in the RAPS model, with an AUROC of 0.601. The BS for the NEWS, qSOFA, EWS, WPS, RAPS, and REMS was 0.18, 0.09, 0.03, 0.14, 0.15, and 0.11 respectively. Excellent calibration was obtained for the NEWS, while the other models had proper calibration. Conclusion: The WPS, REMS, and NEWS have a fair discriminatory performance and may assist in risk stratification for SARS-COV2 patients presenting to the ED. Generally, underlying diseases and most vital signs are positively associated with mortality and were different between the survivors and non-survivors. How to cite this article: Rahmatinejad Z, Hoseini B, Reihani H, Hanna AA, Pourmand A, Tabatabaei SM, et al. Comparison of Six Scoring Systems for Predicting In-hospital Mortality among Patients with SARS-COV2 Presenting to the Emergency Department. Indian J Crit Care Med 2023;27(6):416-425.

Comparing In-Hospital Mortality Prediction by Senior Emergency Resident's Judgment and Prognostic Models in the Emergency Department.

Background: A comparison of emergency residents' judgments and two derivatives of the Sequential Organ Failure Assessment (SOFA), namely, the mSOFA and the qSOFA, was conducted to determine the accuracy of predicting in-hospital mortality among critically ill patients in the emergency department (ED). Methods: A prospective cohort research was performed on patients over 18 years of age presented to the ED. We used logistic regression to develop a model for predicting in-hospital mortality by using qSOFA, mSOFA, and residents' judgment scores. We compared the accuracy of prognostic models and residents' judgment in terms of the overall accuracy of the predicted probabilities (Brier score), discrimination (area under the ROC curve), and calibration (calibration graph). Analyses were carried out using R software version R-4.2.0. Results: In the study, 2,205 patients with median age of 64 (IQR: 50-77) years were included. There were no significant differences between the qSOFA (AUC 0.70; 95% CI: 0.67-0.73) and physician's judgment (AUC 0.68; 0.65-0.71). Despite this, the discrimination of mSOFA (AUC 0.74; 0.71-0.77) was significantly higher than that of the qSOFA and residents' judgments. Additionally, the AUC-PR of mSOFA, qSOFA, and emergency resident's judgments was 0.45 (0.43-0.47), 0.38 (0.36-0.40), and 0.35 (0.33-0.37), respectively. The mSOFA appears stronger in terms of overall performance: 0.13 vs. 0.14 and 0.15. All three models showed good calibration. Conclusion: The performance of emergency residents' judgment and the qSOFA was the same in predicting in-hospital mortality. However, the mSOFA predicted better-calibrated mortality risk. Large-scale studies should be conducted to determine the utility of these models.

Internal Validation of the Predictive Performance of Models Based on Three ED and ICU Scoring Systems to Predict Inhospital Mortality for Intensive Care Patients Referred from the Emergency Department.

Background: A variety of scoring systems have been introduced for use in both the emergency department (ED) such as WPS, REMS, and MEWS and the intensive care unit (ICU) such as APACHE II, SAPS II, and SOFA for risk stratification and mortality prediction. However, the performance of these models in the ICU remains unclear and we aimed to evaluate and compare their performance in the ICU. Methods: This multicenter retrospective cohort study was conducted on severely ill patients admitted to the ICU directly from the ED in seven tertiary hospitals in Iran from August 2018 to August 2020. We evaluated all models in terms of discrimination (AUROC), the balance between positive predictive value and sensitivity (AUPRC), calibration (Hosmer-Lemeshow test and calibration plots), and overall performance using the Brier score (BS). The endpoint was considered inhospital mortality. Results: Among the 3,455 patients included in the study, 54.4% of individuals were male (N = 1,879) and 26.5% deceased (N = 916). The BS for the WPS, REMS, MEWS, APACHE II, SAPS II, and SOFA were 0.178, 0.165, 0.183, 0.157, 0.170, and 0.182, respectively. The AUROC of these models were 0.728 (0.71-0.75), 0.761 (0.74-0.78), 0.682 (0.66-0.70), 0.810 (0.79-0.83), 0.767 (0.75-0.79), and 0.785 (0.77-0.80), respectively. The AUPRC was 0.517 (0.50-0.53) for WPS, 0.547 (0.53-0.56) for REMS, 0.445 (0.42-0.46) for MEWS, 0.630 (0.61-0.65) for APACHE II, 0.559 (0.54-0.58) for SAPS II, and 0.564 (0.54-0.57) for SOFA. All models except the MEWS and SOFA had good calibration. The most accurate model belonged to APACHE II with lowest BS. Conclusion: The APACHE II outperformed all the ED and ICU models and was found to be the most appropriate model in predicting inhospital mortality of patients in the ICU in terms of discrimination, calibration, and accuracy of predicted probability. Except for MEWS, the rest of the models had fair discrimination and partially good calibration. Interestingly, although the REMS is less complicated than the SAPS II, both models exhibited similar performance. Clinicians can utilize the REMS as part of a larger clinical assessment to manage patients more effectively.

Internal validation and evaluation of the predictive performance of models based on the PRISM-3 (Pediatric Risk of Mortality) and PIM-3 (Pediatric Index of Mortality) scoring systems for predicting mortality in Pediatric Intensive Care Units (PICUs).

PURPOSE: The study was aimed to assess the prognostic power The Pediatric Risk of Mortality-3 (PRISM-3) and the Pediatric Index of Mortality-3 (PIM-3) to predict in-hospital mortality in a sample of patients admitted to the PICUs. DESIGN AND METHODS: The study was performed to include all children younger than 18 years of age admitted to receive critical care in two hospitals, Mashhad, northeast of Iran from December 2017 to November 2018. The predictive performance was quantified in terms of the overall performance by measuring the Brier Score (BS) and standardized mortality ratio (SMR), discrimination by assessing the AUC, and calibration by applying the Hosmer-Lemeshow test. RESULTS: A total of 2446 patients with the median age of 4.2 months (56% male) were included in the study. The PICU and in-hospital mortality were 12.4 and 16.14%, respectively. The BS of the PRISM-3 and PIM-3 was 0.088 and 0.093 for PICU mortality and 0.108 and 0.113 for in-hospital mortality. For the entire sample, the SMR of the PRISM-3 and PIM-3 were 1.34 and 1.37 for PICU mortality and 1.73 and 1.78 for in-hospital mortality, respectively. The PRISM-3 demonstrated significantly higher discrimination power in comparison with the PIM-3 (AUC = 0.829 vs 0.745) for in-hospital mortality. (AUC = 0.779 vs 0.739) for in-hospital mortality. The HL test revealed poor calibration for both models in both outcomes. CONCLUSIONS: The performance measures of PRISM-3 were better than PIM-3 in both PICU and in-hospital mortality. However, further recalibration and modification studies are required to improve the predictive power to a clinically acceptable level before daily clinical use. PRACTICE IMPLICATIONS: The calibration of the PRISM-3 model is more satisfactory than PIM-3, however both models have fair discrimination power.

Colorectal Cancer in North-Eastern Iran: a retrospective, comparative study of early-onset and late-onset cases based on data from the Iranian hereditary colorectal cancer registry.

BACKGROUND: The incidence rate of colorectal cancer (CRC) is increasing among patients below 50 years of age. The reason for this is unclear, but could have to do with the fact that indicative variables, such as tumour location, gender preference and genetic preponderance have not been followed up in a consistent mann er. The current study was primarily conducted to improve the hereditary CRC screening programme by assessing the demographic and clinicopathological characteristics of early-onset CRC compared to late-onset CRC in northeast Iran. METHODS: This retrospective study, carried out over a three-year follow-up period (2014-2017), included 562 consecutive CRCs diagnosed in three Mashhad city hospital laboratories in north-eastern Iran. We applied comparative analysis of pathological and hereditary features together with information on the presence of mismatch repair (MMR) gene deficiency with respect to recovery versus mortality. Patients with mutations resulting in absence of the MMR gene MLH1 protein product and normal BRAF status were considered to be at high risk of Lynch syndrome (LS). Analyses using R studio software were performed on early-onset CRC (n = 222) and late-onset CRC (n = 340), corresponding to patients ≤50 years of age and patients > 50 years. RESULTS: From an age-of-onset point of view, the distribution between the genders differed with females showing a higher proportion of early-onset CRC than men (56% vs. 44%), while the late-onset CRC disparity was less pronounced (48% vs. 52%). The mean age of all participants was 55.6 ± 14.8 years, with 40.3 ± 7.3 years for early-onset CRC and 65.1 ± 9.3 years for late-onset CRC. With respect to anatomical tumour location (distal, rectal and proximal), the frequencies were 61, 28 and 11%, respectively, but the variation did not reach statistical significance. However, there was a dramatic difference with regard to the history of CRC in second-degree relatives between two age categories, with much higher numbers of family-related CRCs in the early-onset group. Expression of the MLH1 and PMS2 genes were significantly different between recovered and deceased, while this finding was not observed with regard to the MSH6 and the MSH2 genes. Mortality was significantly higher in those at high risk of LS. CONCLUSION: The variation of demographic, pathological and genetic characteristics between early-onset and late-onset CRC emphasizes the need for a well-defined algorithm to identify high-risk patients.

Internal validation and comparison of the prognostic performance of models based on six emergency scoring systems to predict in-hospital mortality in the emergency department.

BACKGROUND: Medical scoring systems are potentially useful to make optimal use of available resources. A variety of models have been developed for illness measurement and stratification of patients in Emergency Departments (EDs). This study was aimed to compare the predictive performance of the following six scoring systems: Simple Clinical Score (SCS), Worthing physiological Score (WPS), Rapid Acute Physiology Score (RAPS), Rapid Emergency Medicine Score (REMS), Modified Early Warning Score (MEWS), and Routine Laboratory Data (RLD) to predict in-hospital mortality. METHODS: A prospective single-center observational study was conducted from March 2016 to March 2017 in Edalatian ED in Emam Reza Hospital, located in the northeast of Iran. All variables needed to calculate the models were recorded at the time of admission and logistic regression was used to develop the models' prediction probabilities. The Area Under the Curve for Receiver Operating Characteristic (AUC-ROC) and Precision-Recall curves (AUC-PR), Brier Score (BS), and calibration plots were used to assess the models' performance. Internal validation was obtained by 1000 bootstrap samples. Pairwise comparison of AUC-ROC was based on the DeLong test. RESULTS: A total of 2205 patients participated in this study with a mean age of 61.8 ± 18.5 years. About 19% of the patients died in the hospital. Approximately 53% of the participants were male. The discrimination ability of SCS, WPS, RAPS, REMS, MEWS, and RLD methods were 0.714, 0.727, 0.661, 0.678, 0.698, and 0.656, respectively. Additionally, the AUC-PR of SCS, WPS, RAPS, REMS, EWS, and RLD were 0.39, 0.42, 0.35, 0.34, 0.36, and 0.33 respectively. Moreover, BS was 0.1459 for SCS, 0.1713 for WPS, 0.0908 for RAPS, 0.1044 for REMS, 0.1158 for MEWS, and 0.073 for RLD. Results of pairwise comparison which was performed for all models revealed that there was no significant difference between the SCS and WPS. The calibration plots demonstrated a relatively good concordance between the actual and predicted probability of non-survival for the SCS and WPS models. CONCLUSION: Both SCS and WPS demonstrated fair discrimination and good calibration, which were superior to the other models. Further recalibration is however still required to improve the predictive performance of all available models and their use in clinical practice is still unwarranted.

Prognostic utilization of models based on the APACHE II, APACHE IV, and SAPS II scores for predicting in-hospital mortality in emergency department.

BACKGROUND: This study was designed to evaluate and compare the prognostic value of the APACHE II, APACHE IV, and SAPSII scores for predicting in-hospital mortality in the ED on a large sample of patients. Earlier studies in the ED setting have either used a small sample or focused on specific diagnoses. METHODS: A prospective study was conducted to include patients with higher risk of mortality from March 2016 to March 2017 in the ED of Emam Reza Hospital, northeast of Iran. Logistic regression was used to develop three models. Evaluation was performed in terms of the overall performance (Brier Score, BS, and Brier Skill Score, BSS), discrimination (Area Under the Curve, AUC), and calibration (calibration graph). RESULTS: A total of 2205 patients met the study criteria (53% male and median age of 64, IQR: 50-77). In-hospital mortality amounted to 19%. For APACHE II, APACHE IV, and SAPS II the BS was 0.132, 0.125 and 0.133 and the BSS was 0.156, 0.2, and 0.144, respectively. The AUC was 0.755 (0.74 to 0.779) for APACHE II, 0.794 (0.775 to 0.818) for APACHE IV, and 0.751 (0.727 to 0.776) for SAPS II. The APACHE IV showed significantly greater AUC in comparison to the APACHE II and SAPS II. The graphical evaluation revealed good calibration of the APACHE IV model. CONCLUSION: APACHEIV outperformed APACHEII and SAPSII in terms of discrimination and calibration. More validation is needed for using these models for decision-making about individual patients, although they would perform best at a cohort level.

Predictive performance of the SOFA and mSOFA scoring systems for predicting in-hospital mortality in the emergency department.

BACKGROUND: The Sequential Organ Failure Assessment (SOFA) and modified SOFA (mSOFA) are risk stratification systems which incorporate respiratory, coagulatory, liver, cardiovascular, renal, and neurologic systems to quantify the overall severity of acute disorder in the intensive care unit. OBJECTIVE: To evaluate the prognostic performance of the SOFA and mSOFA scores at arrival for predicting in-hospital mortality in the emergency department (ED). METHODS: All adult patients with an Emergency Severity Index (ESI) of 1-3 in the ED of Imam Reza Hospital, northeast of Iran were included from March 2016 to March 2017. The predictive performance of the SOFA or mSOFA scores were expressed in terms of accuracy (Brier Score, BS and Brier Skill Score, BSS), discrimination (Area Under the Receiver Operating Characteristic Curve, AUC), and calibration. RESULTS: A total of 2205 patients (mean age 61.8 ±â€¯18.5 years, 53% male) were included. The overall in-hospital mortality was 19%. For SOFA and mSOFA the BS was 0.209 and 0.192 and the BSS was 0.11 and 0.09, respectively. The estimated AUCs of SOFA and mSOFA models were 0.751 and 0.739, respectively. No significant difference was observed between the AUCs (P = 0.186). The Hosmer-Lemeshow test did not show that the predictions deviated from the true probabilities. Also, the calibration plots revealed good agreement between the actual and predicted probabilities. CONCLUSION: The SOFA and mSOFA scores demonstrated fair discrimination and good calibration in predicting in-hospital mortality when applied to ED. However, further external validation studies are needed before their use in routine clinical care.